Use analytics engineering and the Snowflake Data Cloud with dbt Cloud from Coalesce 2023

"Snowflake Data Cloud is a platform which enables businesses to connect globally across any type or scale of data…"

- Sagar Kulkarni, Partner Sales Engineer at Snowflake

Mathew Zele, Cloud & ISV Lead from Snowflake, leads this conversation with Vivek Pradhan, Lead Partner at Deloitte, and Sagar Kulkarni, Partner Sales Engineer at Snowflake. They discuss the benefits of different tools for data management and offer advice for organizations looking to improve their analytics capabilities.

The rise of SQL in data transformation

The panel discusses the growing prominence of SQL as the transformation language of choice in data engineering. This was seen as a shift from the era of big data and Apache Hadoop, where Python and Scala pipelines were more common.

"The return of SQL as the transformation language of choice is the first big trend we're seeing," says Vivek. He points out that many data teams realized they were essentially executing relational logic, and SQL was generally all they needed.

He also highlights dbt as the central orchestration and transformation layer of choice, regardless of the compute engine used. This trend underscored the alignment between Snowflake and dbt, both of which use SQL at their core.

The move towards federated data infrastructure

The panel also touches on the growing trend of federating data infrastructure. The concept of a single, monolithic data platform was increasingly seen as impractical, leading to greater consideration of a more distributed approach."The idea of federating your data infrastructure is gaining momentum, so I think the trend is growing," says Vivek. However, he notes that the maturity of tooling around data mesh was still an obstacle.

Vivek adds that while many teams want to implement a data mesh, the tooling and engineering workbench are not quite there yet. However, he saw the announcements at Coalesce 2023 as a significant step forward in making data mesh more practical for data teams.

The importance of good engineering discipline, the modern data stack, and simplicity

The panel emphasizes three key factors for successful data and analytics teams: establishing good engineering discipline, embracing the modern data stack, and trading complexity for simplicity.

Vivek highlights that good engineering discipline is key to achieving speed and scale. "If you want speed with scale, you're going to have to have the right engineering disciplines in place," he says. He also urges teams to embrace the modern data stack, which often includes dbt and Snowflake, and to trade unnecessary complexity for technical simplicity. "It takes extra skill to keep things simple, so trading simplicity over complexity is the third [piece of] advice I’d give to data teams."

Lastly, Sagar from Snowflake emphasizes the significance of choosing the right tooling by saying, "It should be dbt and Snowflake full stop." This highlights the confidence the panel has in these tools to deliver value to teams.

Insights surfaced

• SQL is returning as the transformation language of choice for many organizations
• The concept of a federated data infrastructure is gaining momentum
• Good engineering discipline, the adoption of the modern data stack, and trading complexity for simplicity are key to successful data and analytics teams
• Starting small and gradually expanding is a successful strategy for analytics projects
• Snowflake and dbt can work together effectively to reduce complexity and help organizations become exponential enterprises